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11.2.4 Classification of States 

To better understand Markov chains, we need to introduce some definitions. The first 
definition concerns the accessibility of states from each other: If it is possible to go from state 
i to state j, we say that state j is accessible from state i. In particular, we can provide the 
following definitions. 

(n) 

We say that state j is accessible from state i, written as i —> j, if p.- > 0 for some 71. We 
assume every state is accessible from itself since p^ — 1. 

Two states i and j are said to communicate, written as i j, if they are accessible from 
each other. In other words, 


i <->• j means i —> j and j —>• i. 

Communication is an equivalence relation. That means that 

— every state communicates with itself, i -O- i\ 

— if i -O- j, then j i; 

— if i <-> j and j -O- k, then i -O k. 

Therefore, the states of a Markov chain can be partitioned into communicating classes such 
that only members of the same class communicate with each other. That is, two states i and j 
belong to the same class if and only if i j. 


Examplell.6 

Consider the Markov chain shown in Figure 11.9. It is assumed that when there is an arrow 
from state i to state j, then p^j > 0- Find the equivalence classes for this Markov chain. 
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• Solution 

o There are four communicating classes in this Markov chain. Looking at Figure 
11.10, we notice that states 1 and 2 communicate with each other, but they do not 
communicate with any other nodes in the graph. Similarly, nodes 3 and 4 
communicate with each other, but they do not communicate with any other nodes 
in the graph. State 5 does not communicate with any other states, so it by itself is 
a class. Finally, states 6, 7, and 8 construct another class. Thus, here are the 
classes: 


Class 1 
Class 2 
Class 3 
Class 4 


{state 1, state 2}, 

{state 3, state 4}, 

{state 5}, 

{state 6, state 7, state 8}. 
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A Markov chain is said to be irreducible if it has only one communicating class. As we will 
see shortly, irreducibility is a desirable property in the sense that it can simplify analysis of the 
limiting behavior. 

A Markov chain is said to be irreducible if all states communicate with each other. 

Looking at Figure 11.10, we notice that there are two kinds of classes. In particular, if at any 
time the Markov chain enters Class 4, it will always stay in that class. On the other hand, for 
other classes this is not true. For example, if Xq — 1, then the Markov chain might stay in 
Class 1 for a while, but at some point, it will leave that class and it will never return to that 
class again. The states in Class 4 are called recurrent states, while the other states in this chain 
are called transient. 

In general, a state is said to be recurrent if, any time that we leave that state, we will return to 
that state in the future with probability one. On the other hand, if the probability of returning is 
less than one, the state is called transient. Here, we provide a formal definition: 

For any state i, we define 

fa — P(X n — i , for some n > 1\Xq — i). 

State i is recurrent if fa — 1, and it is transient if fa < 1. 

It is relatively easy to show that if two states are in the same class, either both of them are 
recurrent, or both of them are transient. Thus, we can extend the above definitions to classes. 

A class is said to be recurrent if the states in that class are recurrent. If, on the other hand, the 
states are transient, the class is called transient. In general, a Markov chain might consist of 
several transient classes as well as several recurrent classes. 

Consider a Markov chain and assume Xq — i. If i is a recurrent state, then the chain will 
return to state i any time it leaves that state. Therefore, the chain will visit state i an infinite 
number of times. On the other hand, if i is a transient state, the chain will return to state i with 
probability < 1. Thus, in that case, the total number of visits to state i will be a 
Geometric random variable with parameter 1 - fii- 

Consider a discrete-time Markov chain. Let V be the total number of visits to state i. 
a. If i is a recurrent state, then 


P{V = oo|X 0 = i) = 1. 


b. If i is a transient state, then 

F|V 0 = i ~ Geometric{ 1 — fa). 
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Example 11.7 

Show that in a finite Markov chain, there is at least one recurrent class. 

• Solution 

o Consider a finite Markov chain with T states, S = (1,2, Suppose that 

all states are transient. Then, starting from time 0, the chain might visit state 1 
several times, but at some point the chain will leave state 1 and will never return 
to it. That is, there exists an integer M\ > 0 such that X n 7 ^ 1, for all 
71 > M\ . Similarly, there exists an integer M 2 > 0 such that X n 7 = 2, for 
all n > M 2 , and so on. Now, if you choose 

n > max{Mi, M2, • • •, M r }, 

then X n cannot be equal to any of the states 1,2, * * - , 7 *. This is a contradiction, 
so we conclude that there must be at least one recurrent state, which means that 
there must be at least one recurrent class. 


Periodicity: 

Consider the Markov chain shown in Figure 11.11. There is a periodic pattern in this chain. 

(n) 

Starting from state 0, we only return to 0 at times 71 — 3, 6, • • \ In other words, Pqq = 0, 
if 71 is not divisible by 3. Such a state is called a periodic state with period <i(0) = 3. 

1 


1 


Figure 11.11 - A state transition diagram. 
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(n) 

The period of a state 2 is the largest integer d satisfying the following property: p-- = 0, 

whenever 71 is not divisible by d. The period of 2 is shown by d(i) . If p'-- ^ — 0, for all 
71 > 0, then we let d{l) — OO. 


-lfd(i) > 1, we say that state 2 is periodic. 
-lfd(i) = 1, we say that state 2 is aperiodic. 


You can show that all states in the same communicating class have the same period. A class is 
said to be periodic if its states are periodic. Similarly, a class is said to be aperiodic if its states 
are aperiodic. Finally, a Markov chain is said to be aperiodic if all of its states are aperiodic. 

If 2 j, then d(i) — d(j). 

Why is periodicity important? As we will see shortly, it plays a roll when we discuss limiting 
distributions. It turns out that in a typical problem, we are given an irreducible Markov chain, 
and we need to check if it is aperiodic. 

How do we check that a Markov chain is aperiodic? Here is a useful method. Remember that 
two numbers 771 and l are said to be co-prime if their greatest common divisor (gcd) is 1 , i.e., 

gcd(Z, 77l) — 1 . Now, suppose that we can find two co-prime numbers l and 771 such that 
(l) (m) 

Pti >0 and pi"-' > 0. That is, we can go from state 2 to itself in l steps, and also in 771 
steps. Then, we can conclude state 2 is aperiodic. If we have an irreducible Markov chain, this 
means that the chain is aperiodic. Since the number 1 is co-prime to every integer, any state 
with a self-transition is aperiodic. 

Consider a finite irreducible Markov chain X n : 

a. If there is a self-transition in the chain (pa > 0 for some 2), then the chain is 
aperiodic. 

b. Suppose that you can go from state 2 to state 2 in l steps, i.e., p^ > 0. Also suppose 

that p'- ^ > 0. If gcd(Z, 772) = 1 , then state 2 is aperiodic. 

c. The chain is aperiodic if and only if there exists a positive integer 71 such that all 
elements of the matrix P n are strictly positive, i.e., 

Pif > 0, for all 2 , j e S. 


Example 11.8 

Consider the Markov chain in Example 11.6 . 


a. is Class 1 

b. is Class 2 

c. is Class 4 


{state 1, state 2} aperiodic? 

{state 3, state 4} aperiodic? 

{state 6, state 7, state 8} aperiodic? 
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• Solution 

o a. Class 1 = {state 1, state 2} is a periodic since it has a self¬ 
transition, P 22 > 0. 

b. Class 2 = {state 3, state 4} is periodic with period 2 . 

c. Class 4 = {state 6, state 7, state 8} is a periodic . For example, 
note that we can go from state 6 to state 6 in two steps (6 — 7 — 6) and in 
three steps (6 — 7 — 8 — 6). Since gcd(2, 3) = 1, we conclude state 
6 and its class are aperiodic. 


<— previous 
next —> 
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